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1 . I ntroducti on 



The objectives of this research were stated in the initial 
proposal, submitted August 5, 1968: 

Skills such as maneuvering and navigation are too complex 
to be taught through standard comp uter- a s s i s t ed instructional 
lessons. The procedures to be taught are richer and more 
varied, and the scope of student learning difficulties much 
greater, than those encountered in teaching simpler skills 
such as stenotypy or telegraph keyset operation. To handle 
these harder problems effectively, an instructional program 
must include procedures enabling the computer to diagnose 
the student's learning difficulties as he tries to carry out 
the tasks being taught. To do this, the computer must be 
given a means of tracking a student's behavior and of 
generating and testing hypotheses about his difficulties. 

We plan to investigate the incorporation of these capabili- 
ties in a computer program called an automated instructional 
monitor . 

We described a three-year program of research on computer- 
based instructional techniques incorporating diagnosis of student 
difficulties in acquiring complex concepts and skills in perform- 
ing operational tasks. The work was to extend earlier research 
with a simple student monitoring program, SIMON, which was able 
to test the validity of a student's programmed procedure for 
performing a task but was not able to diagnose any errors made 
along the way. In the new approach, the computer would automati- 
cally generate a dynamic display representing the simulated task. 
The student would communicate with the computer through the 
display or keyboard in the course of performing the task. As he 
did so, the monitor system would track and analyze the student's 
work. Hypotheses describing specific learning difficulties 
would be tested and appropriate advice given. 

During the first year we sought to build the component 
programs needed for such an instructional monitor. These' inc luded 
programs for representing the training task and situation in a 
suitably realistic simulation, programs for diagnosing trainee 
actions including erratic behaviors, and dialogue programs to 
enable communication between the trainee and the system. We 
planned during the second year to integrate the components into 
an effective monitor system and to apply it in an elaborate 
extended training situation. The final year was to involve 
evaluation of the material produced in an actual training situa- 
tion and investigation of the problems involved in applying the 
monitoring techniques thus far developed to other operational 
training task s . 



The operational training tasks of particular interest to us 
were in the areas of naval maneuvering, instrument flying, and 
navigation. We selected the first two of these areas. The first 
task chosen was a perceptual motor task: maintaining a simulated 
aircraft in flight on a holding pattern over ja prescribed course 
while guided solely by standard flight instrument indicators 
(i.e., without visual sighting of the real scene). The second 
area of work involved those perceptual skills underlying maneuver- 
ing board work: estimating the relative courses of ships moving 

at varied rates and in various configurations. Naval maneuvering 
and collision avoidance skills were the objects of our interest 
her e . 



An instructional monitor for the holding pattern test was 
designed, implemented, and tested with instrument flight trainees. 
This work is described in Section 2. A program for generating 
and presenting maneuvering problems on a simulated PPI screen 
was designed and implemented. Using this, a graduated sequence 
of course estimation problems was presented to a number of subjects. 
By studying and analyzing trainee behaviors, we characterized 
conspicuous learning difficulties and patterns of skill acquisi- 
tion. This work is described in Section 3. In Section 4 we 
discuss implications of the work done in both areas for further 
research and application. 

Several persons collaborated in this study. The task 
simulations, training experiments, and associated computer 
programs were designed by Wallace Feurzeig, George Lukas, 

Patrick McHugh, and Paul Wexelblat. In the holding pattern work, 
the training problems were provided by McHugh, an experienced 
instrument flight instructor. McHugh also supervised the flight 
testing of the trainees. Feurzeig and Wexelblat designed the 
instructional monitors. The programming was done by Wexelblat, 
with some assistance from Philip Faflick and Richard Grant. 

Seymour Papert contributed to the early planning phase of this 
work. In the ship maneuvering work, Lukas designed and 
analyzed the experiment, and Wexelblat wrote the computer 
programs . 

The report was written by Feurzeig and Lukas and edited 
by Pearl Stock well. 
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Instructional Monitors and the Molding Pattern Task 



2 . 



Investigating the use of computers to help teach complex 
skills such as ship maneuvering and instrument flying was the 
overall aim of this project. Our general approach was to design 
a highly task-specific system of programs (called an instructional 
monitor) for following a student's work and diagnosing his diffi- 
culties along the way. Such a monitor requires detailed informa- 
tion on the different kinds of errors possible, procedures for 
diagnosing specific errors in terms of observable local and 
global effects, and associated information about possible reasons 
for errors in faulty procedures or conceptualization to serve in 
taking corrective action. 

Our personal experiences provided a rich source of pertinent 
error diagnostic information for the holding pattern task in 
instrument flight training. We therefore decided to use that 
task as the initial context for carrying out monitor design and 
experimentation . 

2.1 The Real Task 



One of the major tasks in instrument flying is maintaining 
a holding pattern. The basic holding pattern maneuver is quite 
straightforward: the path is a simple oval consisting of a leg 

that is flown for one minute inbound to a radio beacon, a one- 
minute 180° turn to the right, an outbound one-minute leg, 
another one-minute 180° turn to the right, and this pattern is 
repeated. 




ori b 

MlWUT £ 
TORti 



Realistic complications of the maneuver often confuse the 
student. The apparent complexity occurs when he realizes that 
the pattern is rarely laid out neatly east- west or north-south, 
that there is a wind from some direction, possibly changing in 
velocity and direction, and that there is an overriding necessity 
to keep the aircraft flying as well as maneuvering. 

The greatest single problem throughout the student's initia- 
tion into holding patterns is that of keeping his orientation. 

It is exceptionally easy for a beginning student to forget where 
he is in the pattern, what he is trying to do, and what he is to 
do next. Without a good mental picture of this, his efforts will 




7 



- 3 - 



4 



fail. An essential aspect is a proper instrument scan -- avoiding 
disorientation is crucial and a good instrument cross check is a 
key factor in avoiding disorientation. (The standard instruments 
include a gyrocompass, automatic (radio) direction finder (ADF), 
rate-of-turn indicator, altimeter, and clock.) 

To indoctrinate the student, holding patterns should be 
flown visually at first. As the instruction progresses the 
student will realize that his only fixed "landmark" is the radio 
beacon. He cannot see it, but his instruments can. He must have 
a reasonable idea of what instrument indications to expect at 
various points in the pattern before initiating the hold. Given 
this preplanning, the deviations caused by winds are explicable 
and readily assimilated. 

Typical holding pattern errors made by trainees include the 
following . 

1. Failure to pre-plan the entry before arriving at the hold 
area, resulting in disorientation and improper entry. 

2. Holding on the side opposite that for which the clearance 
was issued. 

3. Slowing of the cross check after arriving at the fix and 
subsequent disorientation. 

4. Uncorrected drift pattern due to failure to visualize the 
wind effects. 

5. Failure to start timing the outbound leg when abeam the 
station. 

6. Varying the bank angle on inbound or outbound turns. 

7. Failure to adjust the outbound leg to assure a one-minute- long 
i nbound leg. 

8. Confusion due to forgetting inbound and outbound headings, 
corrected headings, or wind direction and speed. 

9. Inability to visualize probable ground track. 

10. Failure to see the necessity to vary drift corrections when 
they do not work out as expected. 

11. Fixing attention on a single aspect of the problem resulting 
in a general deterioration. 

12. Failure to maintain attitude. 

13. Failure to hold centerline when inbound to the beacon. 
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The following incorrect paths, which are representative of 
early student work, occur as a result of the indicated errors. 



30 SB CONJ) L£Cr 



BEACON 





Inadequate compensation for 
outbound headwind resulting 
in too short inbound leg. 



two Minute lcg 




one minute leg 



Constant bank with wind. 



BE/9 CO* 




Not shortening outbound leg to 
compensate for tailwind on out- 
bound leg, resulting in an overly 
long inbound leg. 



fccACC'N 




No wind correction angle 
on outbound 1 eg . 



BEACON 

* 




Outbound leg too short, 
turn to inbound leg poorly 
p lanned . 
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Constant bank. 



/ 



BEACON 



YtZhCOH 
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WIMP 



I 





Haphazard wind correction. 







Wind correction angle applied 
in wrong direction on out- 
bound leg with wind from the 
east. 




Wind correction angle applied 
in wrong direction on outbound 
leg with wind from the west. 



Confusion after turning in- 
bound and not finding expected 
instrument indications. 

Flying directly from point A 
to the beacon. 




Correct wind correction angle 
but no "double drift" 
correction. Failure to 
realize problem for some time. 




Holding on wrong side with 
proper wind correction angle. 
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2.2 The Simulated Task 



In the simulation, the trainee has the task of "flying a 
vehicle" over a prescribed course on the basis of instrument 
information. The vehicle is represented as a moving point on a 
scope which also displays the appropriately changing instrument 
information. The location of the vehicle, the ground track for 
the course, the instrument indicators, and the trainee's actual 
flight path can all be displayed. (In later phases, only the 
instrument indicators are shown.) The trainee controls the 
vehicle through use of a joystick linked to the computer. The 
problem may be complicated by the introduction of winds, drift on 
the controls, and variations in altitude. All of this is repre- 
sented in a set of programs implemented on an SDS-940 time-shared 
computer system. The main simulation and display programs are: 

1. Instrument generation and display programs 

2. Holding pattern display program 

3. Ground track display program 

4. Joystick control program 

5. Wind program 

6. Altitude drift program 

7. Holding pattern orientation program. 

The Instrument Display Programs 

The display consists of four instruments: altimeter, ADF 

(Automatic Direction Finder), magnetic compass, and rate-of-turn 
indicator. 

The altimeter simply records the distance above an imaginary 
sea level. A built-in pseudo-random drift factor on altitude 
keeps the plane either rising or falling slowly. This does not 
correspond to the workings of a "real" plane, but the additional 
task of maintaining a given altitude against the random drift 
creates distractions like those involved in actual flying. 

The ADF dial provides relative bearing information. It is 
programmed to point towards a fixed radio beacon on one of the 
corners of the holding pattern. Thus, the direction the needle 
points at any time gives the bearing from the aircraft to the 
station (beacon) . 

The student determines his direction of flight through the 
use of the magnetic compass. The direction the needle points is 
the aircraft's absolute bearing. Under heavy cross wind conditions, 
the student must learn to "crab"-to add a corrective slice to his 
compass heading-if he is to offset the effect of winds and stay 
on cour s e . 
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The last instrument on the "panel" is the rate-of-turn 
indicator. This dial tells the pilot how steeply and to which 
side the plane is banking by displaying the rate-of-turn of his 
current bank. There are three marks on the dial corresponding 
to s tandard- rat e left turn, level flight, and standard-rate right 
turn, respectively. As with the compass, the student must learn 
to anticipate the effect of a crosswind on his ground speed 
during a turn and make the appropriate adjustments in his flight 
plan. 

The Holding Pattern/Ground Track Display Programs 

These programs display a stationary bird's-eye view of the 
imaginary holding pattern including the standard check points and 
the actual course of the plane. The extent of the actual flight 
track shown can be varied. The entire course can be shown or 
just contrails indicating the present location. As well, the 
track can be made invisible. After the student's initial indoc- 
trination to the joystick and dials, this display is turned off 
simulating the "under the hood" situation. After the completion 
of a flight, regenerating the display of the entire track is 
useful for providing a quick summary of the student's performance. 

The Joystick Control Program 

The trainee, "flys" the vehicle by operating a joystick device 
with two control axes: 

(1) The forward and back axis controls the altitude 

(2) The left and right axis controls the bank of the plane. 

Positional information from the joystick is converted into 
digital data by the use of an ana 1 og -to -digita 1 converter. In 
addition, a clock interrupt system was implemented to insure 
real-time operation within a time-shared computer environment. 
Thus, computer elapsed time and actual flying time are guaranteed 
equal regardless of the load (number of active users) on the 
computer . 

Control over the mock aircraft is maintained by operator 
manipulation of the joystick. The operator can determine the 
rate of bank, and thus the rate of turn, by moving the joystick 
to the right or left. The rate of bank increases at a speed 
proportional to the throw of the joystick. The rate of climb or 
descent (altitude) is determined by the forward and backward 
movement of the joystick with the center position corresponding 
to level flight. 



Wind Program 



This program permits adding a constant wind from any direction 
and witl) any speed from zero up to the plane's airspeed . F 1 ying a 
holding pattern with a crosswind half the speed of the aircraft 
is a challenging problem for even experienced pilots. Gusts and 
winds of non-constant speed varying around a specified average 
and range, and shifting in direction, can be introduced. 

Altitude Drift Program 

This introduces a ps eudo - random variation around the nominal 
altitude. 

Holding Pattern Orientation Program 



This permits the orientation of the holding pattern to be 
rotated -- the inbound heading can be set to any value from 0° to 
359°. Figure 1 shows the instrument display along with a holding 
pattern having an orientation of 137° off the standard vertical 
presentation. The four instrument indicators, from top to bottom, 
are the altimeter, radio direction finder, compass, and rate-of- 
turn indicator. 




Fig. 1. Display of Instruments 
and Rotated Holding Pattern. 

Figure 2 shows the console equipment - the display, teletype- 
writer, and joystick control. A close-up of the joystick is 
shown in Figure 3. 
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Fig. 2. Console Arrangement 



Fig. 3. Joystick Control 




2 . 3 Operation of the Monitor Programs 

The use of the programs just described is illustrated next 
with a simple flight indoctrination problem. The situation 
starts with standard initial conditions: no wind, inbound head- 

ing of 0°, the aircraft flying due north along path A-B (see Fig. 
4.1). The altimeter points to 500, the altitude assigned for 
this problem, the compass points to north, the ADF places the 
beacon (located at B) directly ahead, and the rate-of-turn 
indicator says the plane is flying level. Under no-wind condi- 
tions, each of the legs (A to B, B to C, C to D, and D to A) 
takes one minute to fly. Figures 4.1 - 4.6 are photographs taken 
at approximately 40-second intervals as a student gains familiar- 
ity with the instruments. The holding pattern and flight path 
are displayed during this phase. The changing path contrasts 
clearly with the fixed pattern. 

As the student proceeds in his flight, the state of the four 
dials and the position of the plane at each clock interrupt 
(usually every 1/2 second) are stored in the computer. When he 
is through, the student may review his flight in its entirety by 
calling on the replay monitor. This program "plays back" the 
flight, displaying both the instruments and the holding pattern 
along with the ground track. 

As well as reproducing the flight path and all associated 
instrument readings, the replay monitor attempts to detect the 
trainee's local errors and make appropriate comments. The 
trainee starts the replay monitor after carrying out his flight. 
The monitor regenerates his flight, stopping at various points 
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Pig. 4. Early Flight Indoctrination Sequence 
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along the way to make error comments. Each time it makes a 
comment, the monitor freezes the display -- the trainee resumes 
operation after he has noted the comment and studied the display. 

The operation of the replay monitor is illustrated using the 
flight sequence shown in Fig. 4. Figure 4.1 shows the initial 
view seen by the trainee. Within about five seconds of starting, 
the trainee veered off course by initiating a bank towards the 
right. The monitor interrupted with the comment 

NOTE RATE-OF-TURN INDICATOR 
YOU SHOULD NOT HAVE BANKED RIGHT 

Figure 4.2 shows the trainee taking corrective action by 
banking to the left. But, he has overcorrec t ed . Before the 
trainee initiated the sharp left which got him back to the beacon 
(Fig. 4.3), the monitor stopped the action with the comment 

YOU SHOULD HAVE CONTINUED BANKING TO LEVEL OFF 

The trainee has just made it back to the beacon but, in doing so, 
he has left himself in very poor position to initiate his bank 
towards the right for the top loop of the pattern. He should now 
be into his right bank. 

The trainee, realizing that his right turn is too wide (Fig. 
4.4) tries to compensate by tightening up his turn so that he 
will come out at Point C, but he over-adjusts, and comes out too 
far inside. The monitor comments: 

ACCORDING TO THE COMPASS YOU ARE HEADING SOUTHWEST 

THE ADF TELLS YOU THAT THE BEACON IS AHEAD OF YOU AND TO YOUR 

RIGHT 

TOO TIGHT BANK 

In the straight leg C - D, the trainee strays somewhat off 
the pattern. Figure 4.5 shows that he has passed point D and 
still has not turned. This time, as well as commenting on the 
state of the instrument indicators, the monitor declares: 

CORRECTIVE ACTION REQUIRED AT THIS TIME 

As Fig. 4.6 shows, the trainee took appropriate action by 
making a good right turn which is leading right back to the 
starting point A. 

The replay monitor works by projecting the trainee's current 
actions to assess likely outcomes. It attempts to recognize 
aberrant actions like making a bank in the wrong direction or at 
the wrong time. But, it does not analyze these actions globally 
to try to describe the trainee's entire performance. To help 
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characterize the trainee's underlying difficulties in terms of 
standard holding pattern errors, we designed another program, an 
analytic monitor. Like the replay monitor, the analytic monitor 
is used after the termination of a training session. Its opera- 
tion is, however, quite different. It carries out a post-mortem 
of the flight in which it tries to match the pattern of actual 
errors made by the trainee throughout the sequence with the 
characteristic patterns describing standard holding pattern 
errors like those listed in Section 2.1. 

The operation of the analytic monitor is illustrated with an 
actual example in a problem incorporating winds. Figure 5 shows 
photographs from the flight sequence. The instructor, in setting 
up the problem, called the wind program. He gave it three inputs 
which specified the wind type as constant, with a 90° heading 
(i.e., our of the west) and a velocity of 10 knots, a moderate 
level (the speed of the plane is 100 knots). 

A student then flew the sequence excerpted in Figs. 5.1 - 
5.6. Its main features are as follows. In Fig. 5.1, a few 
seconds after the flight started, we see the effects of the wind 

on the aircraft. The compass shows that the plane is still head- 
ing due north but we see from the flight track that the plane has 

been blown to the east by the wind. The RDF shows that the beacon 

is no longer directly ahead, due to this wind displacement. The 
trainee initiates a right bank to compensate for the wind and 
(Fig. 5.2) makes it to the beacon although he was off course. 

At Point B the trainee had initiated a standard 2-minute 
turn. But, because he failed to account for the wind, the turn 
is much too wide (Fig. 5.3), and he is off course again. As can 
be seen in Fig. 5.4, the pilot continues his turn, attempting to 
get himself back into the screen. 

He now (Fig. 5.5) comes well into the pattern, possibly over- 
compensating because he realizes that the next turn will need to 
be tighter because lie will be turning into the wind. Figure 5.6 
shows that he waited a little too long to initiate this turn and, 
thus, went out too far. But he seems to be catching on. 

He should note that the radii of the two turns were different 
due to the wind even though both were made with the same nominal 
rate of turn. These differential wind effects would, of course, 
have been amplified if the wind had been stronger. 

4 

At this point the flight session was terminated. The 
analytic monitor was invoked and it typed out the following list 
of apparent errors. 








0 



Fig. 5. 



Flight Sequence With Moderate Winds 
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NEGLECTING CROSSWIND EFFECT ON GROUND SPEED DURING OUTBOUND LEG 
HOLDING ON SIDE OPPOSITE THAT FOR WHICH CLEARANCE WAS ISSUED 
FAILURE TO VISUALIZE EFFECT OF WIND ON PATTERN 
FAILURE TO VISUALIZE THE DIFFERENTIAL EFFECT OF WIND INBOUND 
AND OUTBOUND 



In more extended sequences the trainee often compounds errors 
of these kinds. Such effects can be noted by the monitor. In 
the flight track shown in Fig. 6, for example, the trainee made 
two full circuits of the pattern. The effects that earlier errors 
had in creating subsequent difficulties were noted by the monitor. 
Also, it commented on failures to make use of previous knowledge. 




Fig. 6. A Two-Circuit Sequence 



For example, on the first circuit the trainee correctly noted the 
drift due to the wind on the outbound leg as shown by his making 
the necessary compensation (the "double drift correction") on the 
inbound leg. However, he did not follow through with the proper 
wind correction the second time around (note the convex A to B 
arc) and the monitor commented on this failure. 

The analytic monitor needs two data inputs. One of these is 
a table of the plane's locations at uniform times throughout the 
sequence. This actual flight position table is generated during 
the flight session and also can be used by the replay monitor to 
reproduce the flight sequence on the display. The other input is 
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a set of tables, one for each characteristic error in the list of 
standard holding pattern errors. Each of these characteristic 
error tables specifies an interval of allowable vehicle positions 
as a function of time, winds, pattern orientation, and desired 
accuracy. For example, the characteristic error of neglecting 
the effect of crosswind on ground speed is described in a table 
which gives the vehicle's approximate position in space every 
thirty seconds of flight, including the effect of crosswind speed 
and direction. The internal operation of the analytic monitor, 
then, is as follows. The program scans the entries in the 
trainee's actual flight position table and compares these with 
the corresponding t im e - p o s i ti o n data generated from the character- 
istic error tables, to determine if there is an acceptable match. 
If so, the characteristic errors so identified are listed, as in 
the example above. 

2 . 4 Testing and Eval u a t i o n 



The holding pattern simulation and monitoring system was 
tested and evaluated in an instructional experiment described in 
this section. Our objective was to show that the use of monitor- 
ing techniques could enhance learning of complex perceptual motor 
tasks and thus provide a means of effectively extending the scope 
of useful application of computer assisted instruction. Our 
strategy was straightforward. We used the monitor system to 
train a number of subjects to "fly" holding patterns (at the 
computer) with some degree of proficiency. We then checked 
whether this skill carried over to the real task, by actual flight 
testing of the subjects. 

Four subjects were involved in the experiments. They had 
flying experience spanning a wide range of hours. The least 
experienced subject was a neophyte -- he had no previous flight 
training . The most experienced one had about 200 hours of visual 
flight time, though no instrument flying time. (He has since 
gone on to obtain an instrument flight rating and then a license 
as a flight instructor for both visual and instrument training.) 

The two other subjects were a beginning student with six hours of 
dual (shared) flight training experience and an intermediate 
student with about 50 hours of flight training experience. 

An instructional sequence, comprising an extensive series of 
holding pattern problems of increasing complexity, was given 
individually to the four subjects. In each case the work was 
carried out over four one-hour sessions at the computer. The first 
session was largely given to familiarize the subject with the 
instrument and flight track displays and the use of the joystick. 
The basic holding pattern problem (unrotated pattern, no winds) 
is to be flown with all displays, then only with the instrument 
display. Following this, a problem with a rotated pattern is 
given and then a return to the basic problem. The second session 



introduces winds. The third and fourth sessions present more 
complicated problems in which the effects of variations in wind 
direction, wind speed, and pattern rotation must be considered 
jointly. 

Throughout the sessions, the monitor diagnostics indicated 
each subject's particular areas of difficulties and thereby 
suggested appropriate situations to be presented in subsequent 
problems. Thus, the amount of time spent with each problem type 
and the pattern of progression to more complicated problems 
varied across subjects. Despite these differences, each took 
about the same overall time at the computer. (Typically each 
session included three significantly different problems and, with 
replay, the minimum time required to run through them all was 
about 30 minutes.) 

As the subject carried out each flight his track was 
recorded. Following the flight, he called the replay monitor to 
rerun his work and review his performance. If a subject felt the 
need for further help, he could invoke the analytic monitor. The 
subject's work was also observed by a proctor. Following each 
flight, the proctor - using preassigned criteria - decided whether 
to proceed with a different problem situation. If more practice 
was deemed necessary, a variation of the current problem was given. 
After the session, the flight instructor who designed the sequence 
and the proctor, who was a novice pilot, briefly discussed the 
subject's work with him. The flight instructor then filled out 
flight log forms to assess and record the subject's progress. 

Also, the effectiveness of the monitors' outputs was judged and 
needs for improvements noted. 

Students adapted very quickly to the simulator environment 
and their progress through the testing sequence was relatively 
direct and uncomplicated. Their comprehension of the basic 
aspects of instrument interpretation was rapid. It is interest- 
ing to note that after the first few flights the display of the 
flight path was considered more of a distraction than a help. 

This was true for all subjects. Each found that, in the case of 
anything more than a simple (no wind or pattern rotation) flight, 
the instruments gave a much more reliable indication of the 
situation than they could obtain by watching the flight path on 
the display. 

The first flight of the second session introduced them to 
wind effects. Subjects were not told in advance from which 
direction the wind was coming, or the force of the winds which 
they were to encounter. Nevertheless, after only a few minutes 
of flight, each subject substantially improved his grasp of wind 
effects. By the end of the second session, three of the four 
subjects had learned to accurately estimate wind speed and 
direction. All subjects were correctly using the technique of 
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double drift compensation for winds during the third session and 
all found the critical comments of the replay monitor pertinent 
and useful during this phase of their training. 

Throughout the four sessions the running of the replay 
monitor after flights was helpful to the subjects. When subjects 
were given the option of going ahead without running the replay 
monitor, they usually wanted to use it. They preferred the 
running "play-by-play" commentary of the replay monitor to the 
global error summary provided by the analytic monitor. It is 
interesting to note that on several occasions during the middle 
sessions, subjects were able to anticipate when the replay moni- 
tor was about to make a comment, and often, what that comment 
would be. During this phase each subject was recognizing his own 
difficulties and learning the appropriate action needed for over- 
coming them. 

The training sessions were conducted in the spring of 1970. 
Three of the four subjects were flight tested in late spring and 
summer of that year. The flight instructor who accompanied the 
subjects and supervised their aerial sessions was a professional 
pilot with many years of experience as an instrument instructor. 
The subjects were each taken aloft and put on a course towards 
the Radio-Direction-Finder beacon at a small airport in the 
Boston area. A hood was placed over the subject's eyes to 
prevent him from looking out of the airplane's windows, while 
allowing full view of the instrumentation in the cockpit. He 
was given the problem of ascertaining when he had reached the 
beacon, and after obtaining the bearing of the runway from a 
standard aeronautical chart, of establishing and maintaining a 
holding pattern for this runway over the beacon. He was then 
given control of the aircraft. 

In his test session, which took approximately an ho.ur, a 
subject was given several flight approaches. Under the circum- 
stances prevailing, a single session was sufficient in each case 
to test whether a subject could proficiently and consistently 
execute holding maneuvers under instrument conditions. The 
instructor reported that all subjects performed exceptionally 
well. They demonstrated facility with the use of the equipment 
and good comprehension of the principles involved in performing 
the holding task under moderate wind conditions. Despite the 
gusty air prevailing in each flight, none of the subjects was 
even significantly off course. They clearly showed that they had 
learned to correctly compensate not only for moderate winds but 
also for gusts changing appreciably both in direction and speed 
throughout the flights. 

In conventional instrument flight training, two-to-four 
hours of pre-flight work is typically required to introduce the 
holding pattern task. Such pre-flight training is not in itself 



sufficient for preparing trainees to successfully perform holding 
pattern maneuvers. Even with a good deal more experience and 
preparation on the ground, instrument trainees usually have sig- 
nificant difficulties in orientation in their first attempts at 
flying holding patterns. To attain a modicum of skill, trainees 
require at least another two-to-four hours of training in the air. 
Thus, the success which our subjects showed in transferring their 
skill from the computer training situation to actual flight was 
no t ewo r thy . 

Our experiment was limited in many ways including sample 
size. Nevertheless, despite the wide range of previous flight 
experience among subjects, their required training times and 
their subsequent performance were strikingly uniform. Each 
subject received four hours of pre-flight training at the computer 
and this amount of time was substantially required in all cases. 
The most advanced subject could not have attained the required 
level of proficiency in much less than three hours of computer 
simulation work and the least advanced subject would not have 
benefited appreciably from more than an additional hour of 
practice at the computer. Also, the three subjects who were 
flight tested performed their tasks in very much the same way. 

They each flew six patterns. The performance of the most advanced 
subject was flawless. The other subjects maintained most of 
their patterns in virtually error-free fashion, with only 
occasional minor mistakes which they speedily noted and corrected. 
The fourth subject, the neophyte trainee, was not tested in flight 
because he did not have the experience needed for the parallel 
task of maintaining the airplane in flight. The instructor 
confidently stated that, if this subject could have flown the 
airplane at all, he would have had immediate success in flying 
holding patterns . 

The effectiveness of the instructional programs used in the 
experiment is evident. These programs, particularly the task 
simulator and the diagnostic monitor, were specifically tailored 
to the holding pattern task. Appropriate instructional programs 
of the same kinds can also be designed for use with many other 
complex perceptual motor tasks. For example, several other tasks 
in the same area of introductory flight training including take- 
off, radio navigation techniques, traffic patterns, instrument 
scanning, and crosswind landing approaches, can be effectively 
treated in this way. Computer simulations of the required degree 
of realism and verisimilitude can easily be programmed. Moreover, 
a considerable knowledge of characteristic errors associated with 
these specific tasks exists. Using . this body of information, 
instructional monitor programs can be developed and evolved 
experimentally to a point where they correctly diagnose the most 
important errors of a particular trainee and help find his under- 
lying difficulties. Instead of pursuing this development, we 
chose to work with a different kind of complex task. 



3 . Instructional Monitors and Maneuvering Concepts 

This area was chosen as a second one which involves complex 
behavior in real time mode, yet it is completely different from 
the first in the specific skills which are brought to bear. In 
the holding pattern task, the subject continuously applies course 
corrections, feedback time being negligible compared to human 
response times. Use of a PPI display, on the other hand, involves 
careful selection and acquisition of pertinent information from a 
very slow moving, complex display. The purpose of observation is 
not direct action, but isolation of potentially interesting con- 
figurations for further computational analysis. A side benefit 
of practicing maneuvering with a simulated PPI display is that 
the student gets a good conceptual framework for dealing with 
relative motions, a critical skill for nearly all maneuvering 
s i tuat ions . 

3 . 1 The Real Task 



The PPI operator is presented with a crowded display from 
which to extract a relatively small amount of important informa- 
tion. The display can include echoes corresponding to a large 
number of ship movements and, in addition, those corresponding 
to stationary objects such as coastlines, buoys, and radar beacons. 
Classification of echoes, particularly those of the latter types, 
is often difficult and is further complicated by the appearance 
of many forms of spurious signals such as those due to sea and 
rain clutter and to multiple echoes. 

Collision avoidance is a particularly important application 
utilizing the PPI display. Here, identification of potentially 
dangerous objects is sought as early as possible after their 
appearance on the screen. This task is very difficult, partly 
because of the obscurations described above, and, to a greater 
degree, because of the extreme slowness of apparent motions. If 
the PPI display is set to a scale of one screen inch correspond- 
ing to one nautical mile of actual distance, a motion of 10 knots 
is seen on the screen as a motion of 1/6" per minute. Although 
precise calculations can only be performed by transferral of the 
situation to a maneuvering board, clearly, skill in making good 
estimates can be very helpful in screening interesting objects, 
with relative bearing in the general direction of own ship. 

Such estimates are particularly important when maneuvers to avoid 
collision are being planned because a change of course involves 
simultaneous consideration of all ships on the screen. 

3 . 2 The simulation 

A student works at a PPI screen, simulated on a graphics 
display, and a teletype terminal, both controlled by a minicomputer 
(a DEC PDP-8L). He is automatically presented with a given series 



of estimation problems involving several different scenarios. An 
introduction to each problem is given on the teletype. The PPI 
sequence is presented. Then the specific question is asked. The 
answer. is sometimes to be typed on the teletype and sometimes to 
be indicated by use of a "joystick" which moves a pointer on the 
simulated PPI screen. The subject's responses are automatically 
recorded for analysis, and he is also given feedback on the 
validity and outcome of his responses. The designer of such a 
problem sequence is given great flexibility in describing the 
format, presentation, and content of each question. In addition, 
he can make any of a large number of control options available 
to the student. 

We next give, in full detail, an illustrative example of 
such an estimation problem via photographs and teletype text. 

It is taken from a sequence of problems which we used with 
students and which is described in detail later. The sequence 
photographs of the PPI screen (Fig. 7) show the developing situa- 
tion. The student is also provided with an acetate overlay with 
five range rings for use with the display. The display is 
relative to own ship, at center, with North upward. The vertical 
dotted line is the heading marker. Numbers on the screen are 
provided, under computer control, to label each of the ships, 
which are shown as closely clustered squares. Explanations are 
enclosed within parentheses. 

TYPE C TO CONTINUE (Typed by computer at termination 

of question #1) 

C (Student's typing) 

QUES #2 (Computer presents 

OBSERVE THE DIRECTION OF MOTION OF SHIP 1 problem referring to 
TYPE C TO START configuration on screen) 



(Concurrent display 
showing own ship at 
center - labeled 0 - 
and five other ships) 





Fig. 7-1. 




c 



(Student types C initiating ship 
movements. Ship numbers no 
longer appear) 




7-6 ' 7-7 

Figs. 7-2 - 7-7. PPI Display at 15, 30, 45 , 60, 75 , and 90 

Seconds . 
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(The display freezes, ship numbers reappear, and the following 
question is simultaneously typed on the teletype.) 

USE THE JOYSTICK TO INDICATE THE DIRECTION OF MOTION OF SHIP 1 
BY POINTING ABOUT AN INCH AHEAD ALONG ITS CURRENT PATH 
TYPE C TO CONTINUE 




Fig. 7-8. 

(Concurrent display) 




Fig. 7-9. 

(Student indicates 
course by positioning 
the + marker controlled 
by the joystick - see 
north of ship 3) 
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(Student types C to continue) 



(The ship motions are continued at 
a ten-fold increase in apparent 
speed to show outcome of situation) 




Fig . 7-10. 

(A time exposure. The 
student sees that his 
estimate was a little 
too far north. He can 
now depress one of two 
switches, either repeat- 
ing this question, or 
going on to the next 
one ) 



QUES #3 

OBSERVE THE DIRECTION OF MOTION OF SHIP 1 (He chooses to 

continue) 

The ship movements in the sequence above are summarized by 
the time exposure given in Fig. 7-11. The brighter segments 
correspond to the slower motions in the first part of the 
question prior to "fast motion". 




Fig. 7-11. 





We next describe the facilities generally available for 
designing a problem sequence. A considerable degree of flexi- 
bility is available both in the situation that can be set up and 
in the question format used. Table 1 contains a list of control 
options available. Each option (except the first) can be used 
in two ways. The lesson designer can use an option under program 
control as in the case of appearance of ship numbers and "freezing" 
of the display in the above example. Alternatively, he can 
specify that an option be under control of the student -- as in 
the choice between advancing to the next problem and repeating 
the current one. (The student also had access to ship numbers, 
being able to turn them on at any time.) Option A is available 
only by means of a switch on the graphics unit and cannot be 
controlled by the computer. Thus, a single question consists of 
a combination of text presented on the teletype, display sequences, 
and a subset of the options described in Table 1. Student re- 
sponse is generally required at one or more places in the problem 
and this is done with the teletype, joystick, and console switches. 
These responses are punched out on paper tape for further analysis. 

The lesson designer can also create a variety of different 
situations for display on the PPI screen. He can specify the 
presence of any number of ships, parameters available to him 
include speed and bearing as well as acceleration and rate of 
turn constants for each ship. Ship echoes can be given any shape 
desired, although these shapes do not change as the display 
progresses. Also, stationary objects such as buoys and shore- 
lines can be specified. 

Table 1. Control Options 



A. The display can be run in storage tube mode so that moving 
ships leave trails. This is useful for familiarization 
purposes. 

B. After completing any problem the student, by depressing 
either of two switches, has the choice of repeating it or 
going to the next one. 

C. The display of heading marker, speed, and bearing can be 
turned on and off by console switches. 

D. Numbers labeling the ships on the display can be turned on 
and off by console switches. 

E. The display can be "frozen" at any point by typing F and 
restarted by typing C on the teletype. 

F. A "joystick" can be used to move a point on the screen, 
either to answer questions or as a variable range marker. 
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G. After a question has been answered, the student can continue 
with a speeded up display to see the outcome of his 
predictions by typing C on the teletype. 

H. Own ship's bearing and speed can be changed by typing any 
of the following 

R followed by Inew bearing (turn right) 

L followed by new bearing (turn left) 

S followed by new speed in knots (change speed) 



The display is relative to own ship. North upwards, and a 
range ring overlay and optional heading marker also are used. 
Figures 8 and 9, which are each composite photographs, show the 
PPI simulation for the case of own ship accelerating and decel- 
erating, maintaining its bearing in each case. 




Fig. 8. Own Ship Proceeding on Course 
000°. Accelerating from 10 Knots. 
Speed number is blurred because it is 
changing. Heading marker has been 
switched off and range rings removed. 




Fig. 9. Own Ship Proceeding on Course 
000°. Decelerating from 10 Knots to 
0 Knots . 



3 . 3 Experimental Set-up 

We settled on the use of a DEC PDP-8L (one of several such 
small computers at BBN) as our research vehicle. This is the 
most satisfactory choice from several points of view. First, 
the PDP-8L is a very inexpensive computer, costing about $10,000, 
complete with teletype and oscilloscope graphics display. This, 
together with the wide availability of this configuration, will 
permit easy reproduction and extension of our work by other 
investigators. We used this computer as a dedicated unit. This 
makes real-time work easy, since there are no "time changes" due 
to system load to contend with. Finally, the computer and 
associated graphics are extremely reliable - there are none of 
the breaking-in and down time problems one encounters with 
sophisticated, state-of-the-art graphics systems. 

Our computer configuration consists of a PDP-8L minicomputer 
with teletype, 4K core, and two programmable internal clocks (an 
option) . An analog-to-digital converter is used to input the 
voltages across a standard resistor joystick. A digital-to- 
analog converter outputs voltages used to plot points on an 
oscilloscope screen. Our choice of oscilloscope, the Tektronix 
611, combines a storage tube, a useful option for familiarization 
with a fairly large screen (7x9 inches). Thus, our simulated 
PPI screen can be 7 inches in diameter. A tape punch is used to 
record student responses. 
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The main computer program is a loop which sweeps a question 
table. The exact format of this table will vary with the format 
of the problems to be posed; as a specific case we take the 
problem sequence described next. There, all of the problems are 
of the same general form. The following quintuple for each entry 
in the question table suffices to prescribe an item. 

Question number 

# of initial ship configuration to be used 
Display time 
Pointer to initial text 
Pointer to question text 

This main loop also punches out student responses on paper tape 
and responds to those of the system options, described earlier, 
which are made available to the student. 

The display is supported by a "ship table" within the computer 
containing complete information about all the ships displayed. 

This includes both their velocity and position vectors as well as 
acceleration characteristics, the last of these for the case in 
which the ship changes its course. The main loop, driven by a 
millisecond clock within the control processor, calls the display 
loop thirty times a second. This display loop sweeps through the 
table of ship values, refreshing the oscilloscope display. 

The ship table is updated once a second to take changes of 
position due to non-zero velocity into account and to change 
velocities in the case of accelerated motion. 

The programs themselves occupy about 1500 words of the 4000 
word core. About 1000 additional words were used for the question 
table, ship table, and associated text and initial configuration 
data in the sample sequence next described. Thus, considerable 
empty space is left in memory for extensions of the system. 

3 . 4 Experimental Problem Sequence 

As a first step in designing an instructional monitor, we 
devised a short sequence of problems. Eleven experimental subjects 
were used, each of whom spent about an hour and a half on the 24 
problems comprising the sequence. Clearly little training is 
possible over such a short time span. The object of this phase 
was, rather, to obtain data for use in designing subsequent 
training programs. These data were used to determine profiles of 
characteristic behavior against which systematic patterns of 
deviation could be contrasted. We first describe the problem 
sequence and then give an analysis of the data generated by its 
use . 
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We designed a sequence which was brief enough to administer 
in a single session (subjects required about 100 minutes to 
complete the sequence in most instances) and, at the same time, 
adequately comprehensive for us to find the characteristic 
behaviors present. Preparing a relatively short problem sequence, 
appropriate for this phase of monitor design, poses special 
problems. In a relatively small number of questions (24 in this 
case) one must vary both type of problem and parameters of each 
type over wide ranges. At the same time, however, the questions 
must be sufficiently similar to make feasible comparisons across 
questions so that effects of such variations can be seen. We 
chose four basic problem types with these desired properties and 
ordered these to form a coherent sequence of increasing difficulty 

The first and simplest type is estimation of bearing of a 
moving ship on the simulated PPI display. Only direction is 
involved here. The second asks the point at which a moving ship 
will intersect own ship's heading marker. Now, both a direction 
and a distance are involved. The third type of problem asks for 
the closest point of approach (CPA) to own ship. The subject 
must now determine two directions - the bearing of the ship of 
interest and, additionally, the perpendicular from the line thus 
estimated to own ship. The fourth and last set of problems asks 
the CPA of one ship to another ship, both in motion relative to 
own ship. Now, for the first time the element of time (or 
equivalently, velocity) is involved since the paths of the two 
ships are not sufficient to solve the problem. The time at which 
each point on each path is traversed is also needed for solution. 
This last problem type is particularly useful in connection with 
the basic skills required for course change estimates, as in 
collision avoidance and maneuvering. All ships used in these 
problems were given constant courses. 

We used six questions of each type and therefore 24 questions 
in all. This allowed us to vary appropriate parameters extensive- 
ly. Six initial ship configurations were prepared to be run 
through for each problem type. By using the same six configura- 
tions repeatedly for the four types, we facilitated comparisons 
across types. Although only one additional ship is required for 
the situations used in problem types one, two, and three, and 
two additional ships for the situation used in the last type, 
each configuration contained six ships to add a reasonable measure 
of complication. The display was uniformly maintained for 90 
seconds in every case. 

In order to draw further distinctions for assessing behavior 
patterns, each group of six questions was further halved. The 
first three questions within each problem type specified the 
ship ( s ) of interest in advance. The latter three only specified 
the ship in question after the simulation was completed. Thus, 
the subject in the latter situations had to "follow" all six 
ships on the simulated PPI. 



Thus, there are four groups of six questions each, each 
group being subdivided into two subgroups of three questions each. 
Below, in Table 2, we give a sample teletype recording for each 
of the four major types of question. 

Table 2. 



Type 1. (Same as example in Section 3.2) 

Ship of interest specified in advance. 

QUES # 2 

OBSERVE THE DIRECTION OF MOTION OF SHIP 1 
TYPE C TO START 

(DISPLAY) 

C 

USE THE JOYSTICK TO INDICATE THE DIRECTION Of MOTION OF SHIP 1 
BY. POINTING ABOUT AN INCH AHEAD ALONG ITS CURRENT PATH 
TYPE C TO CONTINUE 
C 



Type 2. Ship of interest specified in advance 
QUES #9 

OBSERVE THE DIRECTION OF MOTION OF SHIP 2 
TYPE C TO START 

(DISPLAY) 

C 

USE THE JOYSTICK TO INDICATE THE POINT AT WHICH THE PROJECTED 
PATH OF SHIP 2 INTERSECTS THE HEADING MARKER 
TYPE C TO CONTINUE 
C 



Type 3. Ship of interest not specified in advance 
QUES #17 

OBSERVE THE DIRECTION OF MOTION OF EACH SHIP ON THE DISPLAY 
TYPE C TO START 



(DISPLAY) 



C 

INDICATE THE POINT AT WHICH SHIP 1 WILL PASS CLOSEST TO OWN SHIP 
TYPE C TO CONTINUE 
C 
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Type 4. Ships of interest specified in advance 
QUES #20 

OBSERVE THE DIRECTION OF MOTION OF SHIP 4 AND SHIP 5 
TYPE C TO START 



(DI SPLAY) 



C 

INDICATE THE POINT AT WHICH SHIP 4 WILL PASS CLOSEST TO SHIP 5 
TYPE C TO CONTINUE 
C 



3 . 5 E xperimental Findings 

There are special difficulties in judging and analyzing 
answers to the course estimation problems described above. First, 
perceptual difficulties enter significantly by way of greatly 
increasing the width of the subject answer distribution. Such 
perceptual difficulties are very much greater in PPI estimation 
problems than in, say, instrument flying problems because of the 
exceedingly small velocities involved. We will see, later, the 
strong, systematic variations of answer distribution width with 
respect to ship velocity and distance of "correct" answer. To 
further complicate the analysis of data, it must be kept in mind 
that the subject is in a learning situation. This means that 
his performance must be evaluated conditionally in the context 
of his position within the training sequence. 

In the preliminary stage of monitor design, described in 
this part of the report, subject responses were evaluated "by 
hand" after the termination of the test run. The paper tapes, 
which had been punched with the answers during the test runs, 
were processed by another computer program to convert the answer 
data into more useful form. This output was searched by the 
experimenter for significant behavior patterns. In later design, 
all analysis should be done directly by the computer immediately 
after each response. The form of such analysis is described 
below, following a description of our research findings. 

The results of a test run using the 24 problem sequence 
described above were 24 coordinate pairs, each relating to a 
specific ship for each problem. A natural center for the coor- 
dinate system in each case is the position of the ship being 
asked about, at the termination of the display simulation. The 
first six problems involved only a direction estimate with 
respect to this final ship position; the others involved a 
distance estimate as well. 



The responses are effectively represented as polar co.ordinate 
pairs, the first elements being the deviation in distance from 
the exact answer, and the second, the angular deviations. By 
putting answers in this form, we eliminate the absolute position 
and orientation of the answer on the display, and replace the 
arbitrary X, Y coordinate system with a much more natural one. 

The only important parameter variation across questions remaining 
was that of relative difficulty. To eliminate this parameter, we 
found the standard deviation for each coordinate in each problem 
using the data of all the eleven subjects tested. We then 
normalized the answers by dividing each by its standard deviation. 
(Actually, to avoid weighting errors, all this was done in X-Y 
coordinates and converted to relative polar coordinates at the 
last step.) 

Table 3 contains the values of the important parameters for 
each of the 24 questions. These are the velocity of the ship of 
interest (in proportional units), the distance of the exact 
answer from the final position of the ship of interest, and the 
distance and angular standard deviations.* Two of these param- 
eters -- the answer distance, R(ANS), and standard deviation in 
distance, SIGMA(R) -- are not meaningful for the first six 
questions in which only an estimate of angle was required. For 
comparison purposes, it is convenient to use a broader classifi- 
cation of velocity and answer distance. Such a scheme is given 
in Table 4 and is far more convenient than numerical values for 
making rough comparisons. 

The strong dependence of answer accuracy (standard devia- 
tions in the answers) is seen easily from Tables 3 and 4. It is 
immediately apparent that high ship velocities result in more 
precise angular determination (questions 3, 10, 14, 16, 17, 20) 

and that short answer distances result in more precise distance 
estimates (questions 9, 13, 23). One would not expect such a 

relation between SIGMA(R) and VELOCITY or between SIGMA(T), the 
angular standard deviation, and R(ANS) nor do such correlations 
exist . 



★ 

Answers have been carried far beyond the two digits of 
significance due to formatting idiosyncrasies of the programming 
language used (BASIC) . 
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Table 3 . 
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VELOCITY 
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SIGMAC r> 


1 


27 1 .529 






0 . 44059 1 


2 


318.798 






0.375236 


3 


588.313 






0.260307 


4 


192 






0.793861 


5 


1 35.765 






1 .65124 


6 


229.085 






0.622358 


7 


294. 1 56 


441 .38 


199.651 


0.451352 


8 


229.085 


469.553 


572.318 


0.720063 


9 


208 


1 6 


13.9818 


0.727675 


10 


41 1 .359 


640.4 


1 72.726 


0.27607 1 


1 1 


294. 1 56 


429.921 


255. 1 37 


0.52299 


12 


318.798 


419.981 


344. 64 1 


0.84947 


13 


288. 555 


73 . 7564 


80 . 6302 


0.59719 


1 A 


41 1 .359 


585.969 


1 34.702 


0.233788 


1 5 


30 4 


336 


89 . 8969 


0.28532 


16 


56 5 • 685 


384. 666 


84.6023 


0.256388 


17 


41 3.299 


545. 234 


131 .798 


0.255887 


18 


294. 156 


610.94 


1 61 


0.294362 


19 


294. 156 


746.705 


21 7.926 


0.325983 


20 


565.685 


360.624 


1 30.686 


0.392682 


21 


294. 1 56 


374.06 


214.212 


0.70206 


22 


229.085 


440 . 7 1 


125.262 


0.441052 


23 


294. 156 


26 .8701 


34.5435 


0.846489 


2 A 


294. 1 56 


51 3. 408 


122.635 


0.260802 







Table 4. 




- 




VELOCITY 


RCANS) 




1 


MEDIUM 






2 


MEDIUM 






3 


HIGH 






4 


LOW 






5 


LOW 






6 


MEDIUM 






7 


MEDIUM 


HIGH 




8 


MEDIUM 


HIGH 




9 


MEDIUM 


LOW 




10 


HIGH 


HIGH 




11 


MEDIUM 


HIGH 




12 


MEDIUM 


HIGH 




13 


MEDIUM 


LOW 




14 


HIGH 


HIGH 




15 


MEDIUM 


MEDIUM 




16 


HIGH 


MEDIUM 
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VELOCI TY 


R(ANS) 


17 


HIGH 


HIGH 


18 


MEDIUM 


HIGH 


19 


MEDI UM 


HIGH 


20 


HIGH 


MEDI UM 


21 


MEDI UM 


MEDIUM 


22 


MEDI UM 


HIGH 


23 


MEDI UM 


LOW 


24 


MEDIUM 


HIGH 



Table 5 contains processed data for two typical test runs. 
The top row for each contains the sub j ect ' s initials and his 
average distance, angular, and total deviations. To cancel the 
effects of problem complexity, the deviation from correct answer 
of each problem is divided by the standard deviation. The table 
then gives the "normalized" deviations for each question (last 
two columns) as well as the numerical values of the student 
answer relative to the final ship position (R and T) . Data in 
this form are relatively easily used to find patterns of 
deviation . 



JGC 

UUESTI ON 
1 

2 

3 

4 

5 

6 

7 

8 

9 

10 
1 1 
1? 

13 

14 

15 

16 
17 
IB 

19 

20 
21 
22 

23 

24 





Tab 1 e 5 . 


0. 566397 


0.744802 


K 


T 


588 • 80S 


"3.78238 


683.52 


3 . 50036 


441 .81 4 


0.734161 


520 


1 .32209 


446.067 


4. 1 4428 


576.056 


5. 18846 


574.943 


4. 1 5526 


519. 199 


0.588003 


35.7771 


5. 17604 


648.346 


6.00828 


8 1 1 . 123 


5.09656 


91 5.956 


4.26046 


51 .225 


3.81633 


499.472 


0.556902 


281 .823 


0. 113792 


379.389 


3.88224 


562.053 


3.2271 


531.744 


3.92699 


854.354 


5.51765 


270. 703 


4. 1 1619 


293.721 


5.22478 


430.81 3 


4.33188 


40 


3. 14159 


386.161 


6.0532K 



0 *6556 



DEV(rO 


DEVC T ) 




0. 198969 




1 .36692 




5. 78540E-2 




- 7.93950E-2 




1 .34777E-2 




-2.37535 


0.664179 


0.464541 


-0.48158 


-0.568362 


1 .18459 


-0.61 4507 


0. 131343 


- 1 .08843 


1 .26708 


-0.540907 


1 .48051 


1 .52716 


- 1 .03921 


0.313137 


-0.569402 


1 .46256 


-0.369879 


0.285541 


0. 584045 


1 .81275E-3 


0.356528 


1 . 1 4561 


-9 • 44342E-2 


-0.522706 


0.852754 


-0.389035 


-0.475185 


0.83315 


- 5.32 105E-2 


-0.94289 


1.172 


-0.44819 


-2 . 33863E-2 


2.31608E-2 


-0.685454 


2.41 1 57 
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WBW 


0.50897 


0.620045 


0. 564508 




UUEbTIOM 


K 


T 


DEV(H) 


DEVC T) 


1 


848. 1 51 


3. 1 6046 




-1 .21258 


2 


778.018 


3.06955 




0.218812 


3 


605.733 


0.982794 




1.01301 


4 


51 5.985 


1 .051 65 




-0.420055 


5 


600.853 


3. 19488 




-0.561488 


6 


634.073 


0.514451 




0.210258 


7 


397.432 


3.841 49 


-0.224933 


-0.230658 


8 


557.25 


0.683709 


-0. 415095 


-0.435448 


9 


17.8885 


5.81954 


-9. 4821 5E-2 


0.0698164 


10 


624.82 


5. 12372E-2 


-4.85847E-3 


9.29519E-2 


1 1 


521.72 


5. 33442 


0 . 1 32775 


-8.60931 E-2 


12 


400 


3.141 59 


-0.0165755 


0.210026 


13 


96.3328 


3.98575 


-0. 479766 


0.596821 


" 1 A — 


625.281 


6. 401 50E-2 


0.364579 


-0.645702 


1 5 


338.373 


6.1647 


0.259174 


— 0. 528566 


1 6 


403 .822 


4. 12439 


0.872841 


0.946249 


1 7 


560.514 


3. 18442 


0.344848 


0.97884 


18 


649.726 


4.04919 


0. 63837 


-0. 107587 


19 


533.846 


5.94722 


-0.617965 


0.928718 


20 


476.386 


3.9983 


1 .3987 


0. 532928 


21 


475.31 


6. 16509 


0. 794498 


0.396475 


22 


352.817 


5.28781 


0. 549338 


1.7192 


23 


16 


1 .63322 


-0.7181 65 


-1 .7 5876 


24 


490.159 


5.1 6874 


0. 1 62581 


-0.98005 



If a student performs equally well across all problems, 
relative to other students, then his normalized deviations will 
vary in a random manner over his 24 response pairs. This is 
generally not the case, however, and there are a variety of error 
patterns which can be found. One searches for correlations 
between deviations (both in magnitude and sign) and each of the 
fo 1 lowing : 

(A) difficulty of problem as measured by size of standard 
deviation 

(B) speed of ship of interest 

(C) distance of answer from final ship position 

(D) whether or not the ship(s) of interest is specified 
in advance. 

In addition, the following patterns are important. 

(E) learning, as indicated by improvement in score. This 
factor did not enter into the present test runs because of 
their relative brevity. 
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(F) correlation between distance and angular errors 

(G) correlation between absolute angle (direction on screen) 

and deviation of angle. 

If we search the data presented in Table 5 with respect to 
these criteria, we find that for subject JGC there is a simple 
pattern in his angular estimate deviations. He did very much 
better, relatively, on "immediate" problems, that is, ones with 
a small answer distance. This is seen clearly below: 

Low Medium High 

Average Angular Deviation 0.3 0.5 1.0 

There is a similar strong correlation between relative error 
in distance and ship velocity. Thus, JGC works well with small 
distances but has trouble in retention when larger distances are 
involved, either directly as distances or as net ship displace- 
ments. The only significant pattern in subject WBW's answers is 
that he gives considerably poor angular estimates when high ship 
velocities are involved. This, again, indicates a difficulty in 
r etent i o n . 

The experimental findings described above are the result of 
a relatively small-scale preliminary investigation. This means 
that "noise" due to random variations in subject performance is 
much larger than would be the case for a large-scale study. The 
main way in which this noise complicates the analysis is in under- 
estimating the magnitude of relative deviations in responses. 

Part of the standard deviation is due to real perceptual complex- 
ity of the task and is not reducible. A large part, however, is 
due to small sample size. This is clearly seen by the fact that 
even the simplext problems, presenting little in the way of 
perceptual barriers to estimation, gave rise to substantial 
standard deviations. If this latter component is eliminated, or 
substantially reduced, patterns of deviation would appear much 
more sharply in subject responses. Thus, not only would the 
patterns described above be easier to find but the detection of 
more complex ones that are presently obscured would be possible. 
Furthermore, the larger sample size makes possible the use of 
standard statistical tests such as bivariate correlations, and 
single numerical coefficients derived for use in detecting the 
appearance of'patterns of error. This greatly facilitates the 
implementatic n of a completely automatic monitoring program. 

The next step, then, in furthering this development is to 
replicate the experiment using a considerably larger subject 
base to enable more precise statistical measurements. The 
diagnostic procedures used can then be incorporated with the 
simulation to provide an immediate monitoring capability. 
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4 . Summary and Conclusions 



In those kinds of learning situations where the acquisition 
of perceptual or motor skills is integral to a problem-solving 
task, the classical computer assisted instructional framework is 
inadequate. In such situations the student's work must be 
assessed, not on whether he can answer questions about how to do 
the task or even on how well he can describe effective procedures 
for doing it, >ut on whether he can actually do it. Thus, given 
that the task of interest can be reasonably simulated on a computer, 
the CAI procedures needed are instructional monitors like those 
we have been developing. 

r i 2 ) 

In earlier work v ' ' we designed two procedure-oriented 

instructional monitors that tested a student's ability to describe 
complex procedures of many kinds. (These contrast with the task- 
oriented monitors described above which are expressly tailored 
for use with particular tasks.) With these monitors students 
were given a problem-solving task, typically in a mathematics, 
physics, or engineering context. The student was supposed to 
solve the problem by writing a program in an elementary program- 
ming language. In using the more general of these two instruc- 
tional monitors, SIMON, the student tests his program by compar- 
ing its operation for any number of particular inputs he chooses, 
with that of a correct program included by the lesson designer. 

When the student thinks his own program is working properly, or 
when he wants advice about how it is incorrect, he i nf orm s SIMON. 
SIMON tests the student's program and looks for cases where it 
fails. Further, it can inform the student which of his inputs are 
not relevant and whether he has neglected some relevant ones. 

The complex tasks of interest to us in connection with 
maneuvering, navigation, and piloting are more naturally learned 
by doing them than by describing how they should be done. Thus, 
a task-oriented monitor provides a more natural instructional 
framework for these tasks than does a procedure-oriented monitor 
like SIMON. 

Such task-oriented monitors must be given very specific 
information about characteristic performance errors and behavior 
patterns exhibited by trainees carrying out the task. This 
information is highly task - dep enden t -- the observable errors 
shown by trainees learning to fly crosswind holding patterns are 
ostensibly different from those shown in learning crosswind 
landing approaches, even when the underlying difficulties of a 
particular trainee are the same in both situations. Given 
descriptions of specific error patterns, then, the monitor compares 
a particular trainee's performance data with these to detect and 
identify his conspicuous difficulties. The program for doing 
this follows along straightforward and fairly general lines. 

IT SIMON-A Simple Instructional Monitor, W. Feurzeig, P.Wexelblat, 
R.C. Rosenberg, IEEE Trans, on Man-Machine Systems, December 1970; 
also BBN Technical Memorandum, August 1970. 

2. Bond Graphs and ENPORT in Elementary Physics Instruction, 

R.C. Rosenberg, W. Feurzeig, P. Wexelblat, IEEE Trans, on Man- 
Machine Systems, December 1970. 



We have worked on instructional monitors in two very 
different situations. In the work on the instrument flight task 
of maintaining a holding pattern, we started with a knowledge of 
characteristic well-defined trainee behavior patterns. These 
data were of sufficient scope and quality to permit us to write 
programs for detecting and diagnosing individual trainee errors. 

In the work on ship maneuvering, on the other hand, we had to 
build up such knowledge of characteristic behavior and error 
patterns preparatory to implementing monitoring programs for 
training in collision avoidance and other maneuvering tasks of 
interest . 

In the work on the holding pattern task, we demonstrated the 
feasibility and usefulness of the monitoring approach as a train- 
ing tool. Further, we evaluated several other tasks in flight 
training to show the extensibility of this approach to them. In 
particular, we were able to describe characteristic behavior and 
error patterns in programmable form for several critical tasks 
in flight training including instrument scanning, radio naviga- 
tion, and crosswind landing approaches. We think that an intro- 
ductory flight training course could be substantially enhanced 
by a digital simulator and monitor system along the lines 
initiated . 

In the area of ship maneuvering, we performed a preliminary 
experiment designed to elicit characteristic trainee behaviors 
in estimating relative ship motions. The analysis indicated 
that the framework used in that work is appropriate for identify- 
ing these patterns. A substantially extended experiment involv- 
ing about 100 subjects is needed to define the basic patterns 
precisely enough to isolate an individual's errors with high 
confidence. Using these characteristic error data, instructional 
monitors to provide training in basic concepts of ship maneuver- 
ing designed along substantially the same lines as the holding 
pattern monitor, appears feasible. 

Heretofore, most developments of CAI techniques have been 
directed toward purely intellectual training tasks in areas such 
as statistical estimation, organic chemical analysis, or grammatic 
French usage. The techniques we have explored seem richly appro- 
priate for extending CAI to complex tasks involving mechanical 
and perceptual as well as intellectual components. This kind of 
extension will be particularly important in making many areas of 
technical training generally accessible. 
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